Repository navigation
fix(write): flush retained input before dictionary encoding exhausts its budget - #1329
suxiaogang223 wants to merge 1 commit into
Conversation
| a writer; handle the error as a failed write operation. | ||
|
|
||
| For formats that support flushing, a data file also flushes when its own | ||
| retained-input reservation reaches `write-buffer-size`, even if encoding (for |
There was a problem hiding this comment.
[P2] Document the buffer option that controls this flush
The public table writer gets the threshold from CoreOptions::write_parquet_buffer_size(), which reads write.parquet-buffer-size and defaults to 256 MiB. The documented write-buffer-size key is read by the sorted global-index builder and leaves this file-writer threshold at its default.
I reproduced the difference through Table::new_write_builder().with_resources(...) with a 6 MiB budget and distinct 4 MiB dictionary-friendly batches. With the documented key set to 1 MiB, the second write fails with ResourceExhausted. With write.parquet-buffer-size=1 MB, all four batches write, commit, and scan back as 4,096 rows with the expected payload.
Please use write.parquet-buffer-size here and cover the option through the public table writer so the test also exercises configuration wiring.
JeonDaehong
left a comment
JeonDaehong
left a comment
There was a problem hiding this comment.
Append tables pass all four writes here (main fails on the second), but postpone-bucket tables (bucket=-2) still hit this: postpone_file_writer.rs:226 checks only in_progress_size. Same 6 MiB budget, write.parquet-buffer-size=1 mb and 4 MiB dictionary batches: the first write keeps 4113 KiB reserved and the second fails with ResourceExhausted, on this branch and on main. kv_data_file_writer.rs:314 has the same check too, though a bucket=1 PK table already fails on the first write here because the sort buffer and the file writer each reserve the batch, so that one may be separate.
Purpose
Linked issue: Closes #1326.
Related to #911 (partial scope).
Repeated strings can retain several MiB of Arrow input while
producing a small Parquet dictionary. Checking only encoded size can exhaust
the input budget even when flushing the current file would allow writing to
continue. This addresses one part of the resource-governance issue.
Prepared with AI assistance. No throughput measurements are available.
Brief change log
Expose the current format writer's input reservation internally and include it
in the data-file flush threshold. Reuse existing reservation release accounting;
other files and readers sharing the context cannot trigger this file's flush.
Writers without a resource context keep the encoded-size policy.
Tests
Validated with the existing Rust 1.97.0 toolchain on macOS arm64:
cargo +1.97.0 test --locked -p paimon --lib: 4804 passed, 6 ignored.cargo +1.97.0 test --locked -p paimon --test writer_resources_test: 10 passed.batches with a 6 MiB budget and a 1 MiB threshold, value readback and row-group
boundaries, shared-context isolation, and reservation lifetime after failed
flush/close. The existing auto-flush-with-tail test also checks per-file bytes.
cargo +1.97.0 fmt --all -- --checkandgit diff --check: passed.cargo +1.97.0 clippy --locked -p paimon --all-targets -- -D warnings: passed.Not run locally: CI-pinned Rust 1.98.0 checks, the full workspace/Vortex
suite, or the Linux/MSRV matrix. The results above cover the listed commands.
API and Format
Internal change; no public Rust or C API and no storage format change.
Row-group boundaries can change with resource accounting. Formats whose flush
is a no-op do not gain memory reclamation. Single-batch admission failures,
background close overlap and allocations outside reservations remain unchanged.
No spill, automatic retry or peak-memory guarantee is introduced.
Documentation
Document the per-file retained-input threshold in the C writer resource guide.